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The  problem  studied 

The  project  concerns  the  development  of  effective  numerical  methods  for 
the  solution  of  stochastic  control  problems.  In  order  to  focus  on  fundamental 
issues,  the  bulk  of  the  model  applications  are  taken  from  current  problems 
in  high  speed  telecommunications,  but  the  results  are  applicable  in  stochastic 
control  in  general. 

Results 

Two  categories  of  problems  are  of  interest.  The  first  concerns  large  deviations 
methods  for  controlled  queueing  networks  of  the  type  that  arise  in  communi¬ 
cation.  Desired  error  rates  (i.e.,  noise  effects)  are  generally  exceedingly  small, 
cannot  be  estimated  directly,  and  require  the  methods  of  large  deviation  theory. 
Numerical  methods  are  essential  to  get  good  estimates  on  performance,  and  to 
enable  good  designs.  Our  numerical  methods  are  the  only  ones  available  at  this 
time.  The  second  category  of  problems  is  the  optimization  of  applications  which 
can  be  approximated  by  controlled  diffusions,  or  diffusion  or  jump  diffusion  pro¬ 
cesses,  as  is  the  case  for  a  large  number  of  the  telecommunications  problems  of 
interest.  This  approximation  is  possible  for  a  large  number  of  important  prob¬ 
lems  in  the  subject  areas.  Our  Markov  chain  approximation  method  is  the 
current  method  of  choice.  The  original  problem  is  approximated  by  a  com¬ 
putable  problem  which  is  still  physically  close  to  it.  Efficient  algorithms  codes 
have  been  developed  and  tested  on  practical  problems  with  excellent  results. 
More  detail  follows. 

The  first  set  of  problems  studied  involves  the  analysis  of  and  development  of 
numerical  methods  for  communication  and  computer  networks,  where  the  noise 
effects  are  small.  It  is  frequently  necessary  that  the  “error”  rates  of  the  system 
be  quite  small.  An  “error”  might  be  a  blocked  transmission  or  loss  of  a  data 
packet  (say,  with  probability  of  order  10”®  or  smaller).  Direct  calculation  of 
such  probabilities  is  memory  intensive  and  slow,  if  not  impossible.  In  addition, 
such  calculations  are  not  very  informative,  since  they  provide  only  a  mass  of 
numbers  with  little  information  on  how  different  parts  of  the  network  interact. 
The  goal  of  the  analysis  is  often  to  understand  how  errors  can  be  avoided,  and 
for  design  or  regulation  purposes  one  needs  more  information.  Direct  simulation 
in  such  cases  is  also  usually  out  of  the  question,  and  in  any  case  it  would  not 
give  the  qualitative  information  needed  for  design.  Since  experience  dictates 
that  the  probabilities  will  scale  in  an  exponential  fashion  with  respect  to  a  pa¬ 
rameter  that  measures  the  “capacity*’  of  the  system,  e.g,  the  buflfer  sizes,  an 
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asymptotic  (large  deviations)  approach  is  natural.  These  techniques  are  natu¬ 
rally  suited  to  the  difficult  types  of  problems  where  rare  events  are  important, 
and  conventional  intuition  not  very  useful.  The  large  deviations  approxima¬ 
tions  are  defined  in  terms  of  the  solution  to  a  variational  problem,  and  it  is  the 
minimizing  parameter  in  this  variational  problem  that  identifies  the  most  likely 
way  in  which  the  undesirable  event  can  occur.  However,  the  discontinuities  and 
other  nonstandard  features  of  communication  networks  make  the  large  deviation 
analysis  of  such  systems  very  difficult,  and  relatively  few  results  are  available 
for  networks.  These  quatities  are  hard  to  compute  and  numerical  methods  are 
essential.  Their  development  is  based  an  an  alternative  and  powerful  appoach 
to  the  general  theory  of  large  deviations,  which  also  faciltates  the  analysis  of 
the  networks  themselves. 

A  key  ingredient  in  our  approach  to  large  deviations  are  representations 
for  the  normalized  quantities  of  interest.  Since  large  deviation  expectations 
and  probabilities  scale  in  an  exponential  fashion,  it  is  natural  to  phrase  the 
convergence  statement  in  terms  of  the  normalized  logarithm  of  the  quantity  of 
interest.  The  representations  facilitate  this  analysis  significantly,  and  allow  one 
to  substitute  the  well  developed  methods  of  weak  convergence  theory  for  the 
relatively  awkward  exponential  estimates  used  in  standard  approaches.  The 
representation  is  as  the  solution  to  a  variational  problem,  and  in  all  cases  it 
takes  the  form  of  a  small  noise  stochastic  control  problem.  Besides  their  use 
in  the  large  deviation  analysis,  the  representations  are  also  important  because 
they  provide  the  starting  point  for  approximations.  Furthermore,  the  methods 
used  for  the  large  deviations  convergence  can  frequently  be  applied  to  prove 
convergence  of  these  approximations. 

The  paper  [A]  and  the  book  [B]  develop  representations  for  a  wide  variety 
of  stochastic  systems.  [B]  focuses  largely  on  discrete  time  processes  in  a  general 
framework,  while  [A]  is  devoted  to  continuous  time  Markov  models  for  queueing 
networks.  Besides  setting  up  the  representations,  [A]  proves  their  convergence 
under  weak  conditions  and  provides  a  means  for  identifying  the  limit  problem 
variational  problem. 

The  paper  [C]  develops  the  theory  for  networks  in  a  different  direction.  In¬ 
stead  of  laying  the  foundations  for  a  very  general  theory,  this  paper  concentrates 
on  a  particular  non-Markovian  network  model  that  is  currently  of  interest  in 
the  context  of  high-speed  data  networks.  It  is  the  first  and  only  paper  to  date 
that  allows  for  a  complete  large  deviations  analysis  of  bursty  data  sources  in  a 
network  environment.  In  particular,  in  [C]  the  variational  problem  that  charac¬ 
terizes  the  large  deviation  approximation  is  solved  explicitly,  thus  providing  a 
concrete  characterization  of  how  the  interactions  between  different  parts  of  the 
network  lead  to  significant  data  loss.  The  methods  can  be  generalized  to  other 
network  and  data  models. 

Lastly  there  is  the  paper  [D],  which  is  concerned  with  rates  of  convergence 
for  approximations  in  stochastic  and  deterministic  control.  The  paper  presents 
a  simple  method  for  obtaining  rate  of  convergence  which  is  applicable  to  a  wide 
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range  of  approximation  problems.  Heavy  use  is  made  of  representations  of  the 
approximations  as  functionals  of  a  controlled  Markov  chain,  which  of  course 
is  a  by-product  of  the  Markov  chain  approximation  method.  The  method  is 
illustrated  thought  the  presentation  of  a  number  of  examples,  including  finite 
difference  schemes  for  stochastic  and  deterministic  optimal  control  problems 
with  several  different  cost  structures.  A  general  principle  can  be  abstracted, 
and  indeed  the  method  may  be  applied  to  a  variety  of  approximation  problems, 
such  as  the  numerical  approximation  of  nonlinear  PDE  not  a  priori  related  to 
control  theory. 

Concerning  the  second  class  of  problems,  codes  have  been  written,  and  ap¬ 
plied  very  successfully  to  problems  in  telecommunications,  yielding  important 
information  on  the  performance  and  design  which  is  not  available  by  other 
means. 

We  have  developed  domain  decomposition  methods  for  the  numerical  solu¬ 
tion  of  Markov  chain  control  problems  on  large  state  spaces  of  the  type  which 
arise  as  approximations  to  optimal  stochastic  control  problems  with  diffusion, 
jump- diffusion  or  reflected  diffusion  type  models.  These  are  also  applicable  to 
the  degenerate  nonlinear  elliptic  partial  differential  integral  equations  or  varia¬ 
tional  inequalities  of  the  type  which  appear  as  “Bellman”  equations.  Whether 
or  not  the  PDE  has  only  a  formal  meaning,  the  solutions  to  the  approximat¬ 
ing  Markov  chain  problems  converge  to  the  desired  cost  or  optimal  functional. 
Owing  to  many  non  standard  features  in  such  problems  and  to  the  nonlinearity 
of  the  equations  which  are  to  be  solved,  the  decomposition  problems  for  the 
Markov  chain  control  problems  often  require  special  methods.  We  prove  that 
appropriate  adaptations  of  current  decomposition  techniques  converge  under 
conditions  which  are  typical  of  many  classes  of  applications.  Probabilistic  in¬ 
terpretations  of  the  various  algorithms  in  terms  of  functionals  of  Markov  chains 
(controlled  or  not)  are  used.  An  important  application  is  to  problems  in  four  or 
more  dimensions  where,  without  decomposition,  the  size  and  structure  of  the 
state  space  and  associated  data  structures  often  leads  to  very  poor  performance 
on  cache  machines. 
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